{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "13d2b729",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/callbacks/HoneyHiveLlamaIndexTracer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d8b66c",
   "metadata": {},
   "source": [
    "# HoneyHive LlamaIndex Tracer\n",
    "\n",
    "[HoneyHive](https://honeyhive.ai) is a platform that helps developers monitor, evaluate and continuously improve their LLM-powered applications.\n",
    "\n",
    "The `HoneyHiveLlamaIndexTracer` is integrated with HoneyHive to help developers debug and analyze the execution flow of your LLM pipeline, or to let developers customize feedback on specific trace events to create evaluation or fine-tuning datasets from production.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5e47da1",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "612f35ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n",
      " ········\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OpenAI API key configured\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "if os.getenv(\"OPENAI_API_KEY\") is None:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass(\n",
    "        \"Paste your OpenAI key from:\"\n",
    "        \" https://platform.openai.com/account/api-keys\\n\"\n",
    "    )\n",
    "assert os.getenv(\"OPENAI_API_KEY\", \"\").startswith(\n",
    "    \"sk-\"\n",
    "), \"This doesn't look like a valid OpenAI API key\"\n",
    "print(\"OpenAI API key configured\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b565e3ef-61cb-4196-81b3-71b4d724434c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paste your HoneyHive key from: https://app.honeyhive.ai/settings/account\n",
      " ········\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HoneyHive API key configured\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "if os.getenv(\"HONEYHIVE_API_KEY\") is None:\n",
    "    os.environ[\"HONEYHIVE_API_KEY\"] = getpass(\n",
    "        \"Paste your HoneyHive key from:\"\n",
    "        \" https://app.honeyhive.ai/settings/account\\n\"\n",
    "    )\n",
    "print(\"HoneyHive API key configured\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fdd01a48",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c5cb91e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78a29d9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.callbacks import CallbackManager\n",
    "from llama_index.core.callbacks import LlamaDebugHandler\n",
    "from llama_index.core import (\n",
    "    VectorStoreIndex,\n",
    "    SimpleDirectoryReader,\n",
    "    SimpleKeywordTableIndex,\n",
    "    StorageContext,\n",
    ")\n",
    "from llama_index.core import ComposableGraph\n",
    "from llama_index.llms.openai import OpenAI\n",
    "from honeyhive.utils.llamaindex_tracer import HoneyHiveLlamaIndexTracer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6feb252",
   "metadata": {},
   "source": [
    "## Setup LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d22fee33",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.llm = OpenAI(model=\"gpt-4\", temperature=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7cff711-8704-4db9-ba81-8160b7bd1447",
   "metadata": {},
   "source": [
    "## HoneyHive Callback Manager Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a32b984-772e-4832-945e-cb6fc7be9e0b",
   "metadata": {},
   "source": [
    "**Option 1**: Set Global Evaluation Handler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a3b9d22-cd67-4fb5-9785-254e58179a02",
   "metadata": {},
   "outputs": [],
   "source": [
    "import llama_index.core\n",
    "from llama_index.core import set_global_handler\n",
    "\n",
    "set_global_handler(\n",
    "    \"honeyhive\",\n",
    "    project=\"My LlamaIndex Project\",\n",
    "    name=\"My LlamaIndex Pipeline\",\n",
    "    api_key=os.environ[\"HONEYHIVE_API_KEY\"],\n",
    ")\n",
    "hh_tracer = llama_index.core.global_handler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1755516-f8ad-458e-b52f-f7665c023e43",
   "metadata": {},
   "source": [
    "**Option 2**: Manually Configure Callback Handler\n",
    "\n",
    "Also configure a debugger handler for extra notebook visibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "defa9155-daca-4a8f-8ca6-87d1ee98f084",
   "metadata": {},
   "outputs": [],
   "source": [
    "llama_debug = LlamaDebugHandler(print_trace_on_end=True)\n",
    "\n",
    "hh_tracer = HoneyHiveLlamaIndexTracer(\n",
    "    project=\"My LlamaIndex Project\",\n",
    "    name=\"My LlamaIndex Pipeline\",\n",
    "    api_key=os.environ[\"HONEYHIVE_API_KEY\"],\n",
    ")\n",
    "\n",
    "callback_manager = CallbackManager([llama_debug, hh_tracer])\n",
    "\n",
    "Settings.callback_manager = callback_manager"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4a7c101",
   "metadata": {},
   "source": [
    "## 1. Indexing"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "81633478",
   "metadata": {},
   "source": [
    "Download Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d0aa69f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p 'data/paul_graham/'\n",
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1011596",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3d6975c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**********\n",
      "Trace: index_construction\n",
      "    |_node_parsing ->  0.080298 seconds\n",
      "      |_chunking ->  0.078948 seconds\n",
      "    |_embedding ->  1.117244 seconds\n",
      "    |_embedding ->  0.382624 seconds\n",
      "**********\n"
     ]
    }
   ],
   "source": [
    "index = VectorStoreIndex.from_documents(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae4de4a9",
   "metadata": {},
   "source": [
    "## 2. Query Over Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42221465",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**********\n",
      "Trace: query\n",
      "    |_query ->  11.334982 seconds\n",
      "      |_retrieve ->  0.255016 seconds\n",
      "        |_embedding ->  0.247083 seconds\n",
      "      |_synthesize ->  11.079581 seconds\n",
      "        |_templating ->  5.7e-05 seconds\n",
      "        |_llm ->  11.065533 seconds\n",
      "**********\n",
      "Growing up, the author was involved in writing and programming. They wrote short stories and tried their hand at programming on an IBM 1401, using an early version of Fortran. Later, they started programming on a TRS-80 microcomputer that their father bought, creating simple games, a program to predict the flight of their model rockets, and a word processor. Despite their interest in programming, they initially planned to study philosophy in college, but eventually switched to AI.\n"
     ]
    }
   ],
   "source": [
    "query_engine = index.as_query_engine()\n",
    "response = query_engine.query(\"What did the author do growing up?\")\n",
    "print(response, sep=\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c49ff101",
   "metadata": {},
   "source": [
    "## View HoneyHive Traces\n",
    "\n",
    "When we are done tracing our events we can view them via [the HoneyHive platform](https://app.honeyhive.ai). Simply login to HoneyHive, go to your `My LlamaIndex Project` project, click the `Data Store` tab and view your `Sessions`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
